QTM 385 - Experimental Methods

Lecture 17 - Interference

Danilo Freire

Emory University

How are you doing? 😊

Brief recap 📚

Brief recap 📚

  • Spillovers are a common problem in social science research
  • They can be positive or negative, and they can be modelled explicitly in our analysis
  • There are several methods to deal with spillovers, such as multi-level designs, within-subject designs, repeated-measures experiments, and waitlist designs
  • Each design has its advantages and disadvantages, and the choice of design should be based on the research question and the context
  • You can use DeclareDesign to simulate spillover designs and pretest posttest designs
  • The statistical analysis of waitlist designs are a little tricky, but you can use the swCRTdesign package in R to help you with that

Source: Forbes (2022)

Today’s plan 📅

Interference and spillovers

  • Three readings for today:
    • Centola (2010): “The Spread of Behavior in an Online Social Network Experiment”
    • Paluk et al (2021): “Changing climates of conflict: A social network experiment in 56 schools”
    • Gerber and Green (2000): “The Effects of Canvassing, Telephone Calls, and Direct Mail on Voter Turnout: A Field Experiment”

Source: Centola (2010)

Interference

Waitlist designs

Waitlist designs

  • The final type of design we will discuss today is the waitlist design, also known as the stepped-wedge design
  • Their scientific value comes from their ability to track treatment effects among several subjects as they play out over time
  • Waitlists play a diplomatic role because they overcome the problem of withholding treatment from a control group (remember our ethics discussion?)
  • In this design, every subject is treated eventually; random assignment determines when they receive treatment
  • It is a hybrid between a within-subject and a between-subject design, and it is widely used in public health interventions (e.g., vaccine rollouts)

Waitlist designs

Source: Hemming et al (2014)

Example: TV advertising and candidate support

  • Imagine you are a political consultant and you want to test the effect of TV advertising on candidate support
  • Ads are aired during three weeks, and outcomes (support for the gubernatorial candidate as gauged by opinion polls) are assessed at the end of each week
  • Eight media markets are randomly assigned to one of four conditions:
    • Two media markets are randomly assigned to air ads for three weeks starting in week 1
    • Two markets air ads for two weeks starting in week 2
    • Two markets air ads for one week starting in week 3
    • And two markets air no ads at all
  • There are just three relevant potential outcomes:
    • \(Y_{00}\): untreated during preceding and current periods
    • \(Y_{01}\): untreated during the preceding period but treated during the current period
    • \(Y_{11}\): treated in both the preceding and current periods
    • Given the design, we never observe the potential outcome \(Y_{10}\) because media markets never cease to run ads once they start

Advertising waitlist experiment’s random assignments and observed outcomes

Assigned treatment

Market Week 1 Week 2 Week 3
1 01 11 11
2 00 00 01
3 00 01 11
4 00 00 01
5 00 00 00
6 01 11 11
7 00 00 00
8 00 01 11

Observed outcomes

Market Week 1 Week 2 Week 3
1 7 9 4
2 7 5 7
3 1 2 10
4 4 3 10
5 3 3 3
6 10 8 10
7 2 3 4
8 3 1 3

Probabilities of assignment to treatment condition

Treatment Condition Week 1 Week 2 Week 3
Pr(00) 0.75 0.50 0.25
Pr(01) 0.25 0.25 0.25
Pr(11) 0 0.25 0.50

Estimating the immediate effect of TV advertising

  • The immediate treatment effect is \(Y_{01} - Y_{00}\), that is, the effect of being treated in the current period but not in the preceding period
  • We just need to take the numbers from the tables and apply inverse probability weighting again

\[ \begin{aligned} \widehat{E}[Y_{01} - Y_{00}] &= \frac{\frac{7 + 10}{0.25} + \frac{2 + 1}{0.25} + \frac{7 + 10}{0.25}}{\frac{2}{0.25} + \frac{2}{0.25} + \frac{2}{0.25}} \\ &- \frac{\frac{7 + 1 + 4 + 4 + 3 + 2 + 3}{0.75} + \frac{5 + 3 + 3 + 3}{0.50} + \frac{3 + 4}{0.25}}{\frac{6}{0.75} + \frac{4}{0.50} + \frac{2}{0.25}} = 2.72. \end{aligned} \]

Estimating the cumulative effect of TV advertising

  • Finally, we will estimate the cumulative effect of TV advertising, which is \(Y_{11} - Y_{00}\), that is, the effect of being treated in both the preceding and current periods
  • We do the same thing again, but now we consider the \(Y_{11}\) potential outcomes
  • However, we must restrict our attention to the second and third weeks, because this type of treatment cannot occur in the first week

\[ \begin{aligned} \widehat{E}[Y_{11} - Y_{00}] &= \frac{\frac{9 + 8}{0.25} + \frac{4 + 10 + 10 + 3}{0.50}}{\frac{2}{0.25} + \frac{4}{0.50}} \\ &- \frac{\frac{5 + 3 + 3 + 3}{0.50} + \frac{3 + 4}{0.25}}{\frac{4}{0.50} + \frac{2}{0.25}} = 4.13. \end{aligned} \]

Conclusion

  • Spillovers are a common problem in social science research
  • They can be positive or negative, and they can be modelled explicitly in our analysis
  • There are several methods to deal with spillovers, such as multi-level designs, within-subject designs, repeated-measures experiments, and waitlist designs
  • Each design has its advantages and disadvantages, and the choice of design should be based on the research question and the context
  • You can use DeclareDesign to simulate spillover designs and pretest posttest designs
  • The statistical analysis of waitlist designs are a little tricky, but you can use the swCRTdesign package in R to help you with that

And that’s all for today! 🎉

See you next time! 😉